Search for: All records

Enterprise AI Assistants are increasingly deployed in domains where accuracy is paramount, making each erroneous output a potentially significant incident. This paper presents a comprehensive framework for monitoring, benchmarking, and continuously improving such complex, multi-component systems under active development by multiple teams. Our approach encompasses three key elements: (1) a hierarchical ``severity'' framework for incident detection that identifies and categorizes errors while attributing component-specific error rates, facilitating targeted improvements; (2) a scalable and principled methodology for benchmark construction, evaluation, and deployment, designed to accommodate multiple development teams, mitigate overfitting risks, and assess the downstream impact of system modifications; and (3) a continual improvement strategy leveraging multidimensional evaluation, enabling the identification and implementation of diverse enhancement opportunities. By adopting this holistic framework, organizations can systematically enhance the reliability and performance of their AI Assistants, ensuring their efficacy in critical enterprise environments. We conclude by discussing how this multifaceted evaluation approach opens avenues for various classes of enhancements, paving the way for more robust and trustworthy AI systems. 

In real-world phenomena which involve mutual influence or causal effects between interconnected units, equilibrium states are typically represented with cycles in graphical models. An expressive class of graphical models, relational causal models, can represent and reason about complex dynamic systems exhibiting such cycles or feedback loops. Existing cyclic causal discovery algorithms for learning causal models from observational data assume that the data instances are independent and identically distributed which makes them unsuitable for relational causal models. At the same time, causal discovery algorithms for relational causal models assume acyclicity. In this work, we examine the necessary and sufficient conditions under which a constraint-based relational causal discovery algorithm is sound and complete for cyclic relational causal models. We introduce relational acyclification, an operation specifically designed for relational models that enables reasoning about the identifiability of cyclic relational causal models. We show that under the assumptions of relational acyclification and sigma-faithfulness, the relational causal discovery algorithm RCD is sound and complete for cyclic relational models. We present experimental results to support our claim. 

ImportanceAbortion bans may lead to births among those who are unable to overcome barriers to abortion. The population-level effects of these policies, particularly their unequal impacts across subpopulations in the US, remain unclear. ObjectiveTo assess heterogeneity in the association of abortion bans with changes in fertility in the US, within and across states. Design, Setting, and ParticipantsDrawing from birth certificate and US Census Bureau data from 2012 through 2023 for all 50 states and the District of Columbia, this study used a bayesian panel data model to evaluate state-by-subgroup-specific changes in fertility associated with complete or 6-week abortion bans in 14 US states. The average percent and absolute change in the fertility rate among females aged 15 through 44 years was estimated overall and by state, and within and across states by age, race and ethnicity, marital status, education, and insurance payer. ExposureComplete or 6-week abortion ban. Main outcome and MeasuresFertility rate (births per 1000 reproductive-aged females) overall and by subgroups. ResultsThere were an estimated 1.01 (95% credible interval [CrI], 0.45-1.64) additional births above expectation per 1000 females aged 15 through 44 years (reproductive age) in states following adoption of abortion bans (60.55 observed vs 59.54 expected; 1.70% increase; 95% CrI, 0.75%-2.78%), equivalent to 22 180 excess births, with evidence of variation by state and subgroup. Estimated differences above expectation were largest for racially minoritized individuals (≈2.0%), unmarried individuals (1.79%), individuals younger than 35 years (≈2.0%), Medicaid beneficiaries (2.41%), and those without college degrees (high school diploma, 2.36%; some college, 1.58%), particularly in southern states. Differences in race and ethnicity and education across states explain most of the variability in the state-level association between abortion bans and fertility rates. Conclusion and RelevanceThese findings provide evidence that fertility rates in states with abortion bans were higher than would have been expected in the absence of these policies, with the largest estimated differences among subpopulations experiencing the greatest structural disadvantages and in states with among the worst maternal and child health and well-being outcomes.